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Abstract 

There  is  a  supposition  in  net-centric  warfare  that  the  effectiveness  of  a  coalition  will  increase 
with  the  amount  of  shared  data.  This  supposition  presumes  that  the  shared  data  contributes 
positively  to  the  situational  awareness  and  effectiveness  of  the  recipient  platform.  In  the  limiting 
cases,  the  resource  cost  to  the  recipient  platform  may  exceed  the  potential  value  of  the 
contributed  information.  In  the  ideal  situation,  shared  data  at  appropriate  levels  of  refinement 
improves  the  effectiveness  of  the  entire  command  team. 

A  series  of  Virtual  Battle  Experiments  (VBEs)  has  been  initiated  with  which  to  investigate  the 
influence  of  sharing  broadband  passive  sonar  data  and  how  this  influence  changes  with  the  level 
at  which  the  data  is  shared.  The  experiments  use  different  operators  in  multiple  sessions  to  make 
statistically  relevant  measurements.  In  the  baseline  experiment,  VBE  CA-1,  the  shared  data  was 
provided  in  a  purely  visual  format.  Operator  effectiveness  was  evaluated  by  analyzing  the  speed 
and  accuracy  with  which  sonar  track  segments  were  manually  associated.  Also  of  interest  in  this 
experiment  was  the  decision  rationale  that  was  used  by  the  operators  to  associate  or  disassociate 
track  segments.  This  paper  summarizes  the  results  of  this  trial  and  outlines  our  plans  for 
extending  these  experiments. 

1 .  Introduction 

1 . 1  Coalition  Data  Sharing 

There  is  a  supposition  in  net-centric  warfare  that  the  effectiveness  of  a  coalition  will  increase 
with  the  amount  of  shared  data  [1].  This  supposition  presumes  that  the  shared  data  contributes 
positively  to  the  situational  awareness  and  effectiveness  of  the  recipient  platform.  Logically, 
however,  the  marginal  utility  of  additional  data  diminishes  since  the  most  valuable  data  would 
typically  be  chosen  to  be  exchanged  first.  Further,  there  is  a  cost  to  processing  the  received  data 
that  typically  increases  with  the  decreasing  information  content  of  the  additional  received  data. 
There  is,  therefore,  little  benefit  to  exchanging  low-level  data  if  the  data  does  not  contribute  to  an 
improvement  in  situational  awareness  or  if  the  recipient  platform  lacks  the  capability  to  further 
process  the  data.  There  may  be  significant  benefit,  however,  if  the  recipient  platform  is  able  to 
jointly  process  the  received  data  with  organic  data  to  produce  a  more  accurate  or  complete 
operational  picture.  The  issue  then  becomes  a  trade-off  between  the  cost  of  exchanging  and 
processing  the  data  and  the  relative  improvement  in  the  operational  picture. 
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Figure  1.  The  sonar  data  refinement  process  requires  some  trade-offs.  The  information  density  of  the  target  data 
increases  as  the  data  is  processed  while  information  that  is  not  considered  to  be  relevant  or  useful  in 
subsequent  stages  is  discarded. 


In  order  to  investigate  the  potential  benefits  of  coalition  data  exchange  it  is  useful  to  examine  the 
results  of  a  number  of  scenarios  in  which  data  at  differing  levels  of  processing  are  shared,  and 
evaluate  the  relative  value  of  the  shared  data.  By  focussing  on  relative  comparisons  between 
specific  scenarios,  statistically  meaningful  measurements  can  be  made  that  predict  the  potential 
value  of  additional  shared  data  in  similar  circumstances.  The  goal  of  this  investigation  is  to 
provide  some  criteria  by  which  to  evaluate  a  suitable  balance. 

1 .2  Sonar  Data  Development 

Passive  sonar  can  be  used  in  the  underwater  environment  to  covertly  detect  and  track  vessels  by 
their  own  acoustic  emissions  [2],  As  shown  in  Figure  1,  there  are  several  steps  in  the  process  of 
refining  passive  sonar  data  from  a  towed  array  receiver  into  estimates  of  target  position,  course 
and  speed  [3],  Data  is  initially  available  from  the  array  as  a  time  series  of  acoustic  pressures 
from  each  hydrophone  in  the  array.  Onboard  the  towing  vessel,  the  measured  pressures  are 
processed  into  time  series  of  received  power  versus  bearing,  which  are  then  presented  to  a  sonar 
operator.  To  an  operator  observing  a  time-bearing  display,  the  acoustic  emissions  of  a  target 
vessel  will  appear  as  an  intensity  peak  changing  slowly,  if  at  all,  in  bearing  over  time.  The  series 


of  intensity  peaks  can  be  assembled  into  a  track  by  a  signal  following  algorithm  and  future 
values  predicted,  assuming  that  there  are  no  rapid  changes  in  the  operation  of  the  target  vessel. 
Should  a  signal  follower  be  unable  to  maintain  lock  on  a  track,  the  track  is  terminated  and  a  new 
track  segment  initiated  when  the  series  of  intensity  peaks  reappears. 

Although  the  passive  sonar  system  does  not  provide  target  range,  course  or  speed  information 
directly,  the  sonar  operator  can  estimate  this  information  by  using  tools  such  as  Target  Motion 
Analysis  (TMA)  to  analyse  the  target’s  track  history.  The  accuracy  of  TMA  is  improved  by  the 
increased  duration  of  the  available  target  track.  In  order  to  increase  the  duration  of  available 
track  history,  the  sonar  operator  may  associate  multiple  sonar  track  segments  that  are  believed  to 
have  originated  from  the  same  target  into  a  longer,  fused  track.  This  task,  however,  can  be 
especially  challenging  given  the  complexities  of  acoustic  propagation  in  the  underwater 
environment.  In  each  of  the  above  stages  of  data  refinement,  the  information  density  of  the 
target  data  increases  as  the  data  is  processed.  Information  that  is  not  considered  to  be  relevant  or 
useful  to  subsequent  stages  is  discarded.  This  results  in  reduced  opportunities  to  pursue  alternate 
tactical  picture  development  paths. 

Data  sharing  can  occur  at  any  stage  in  the  sonar  processing  sequence  described  above.  However, 
the  TMA  results  are  the  most  concise  and,  to  a  recipient  lacking  tools,  staffing  or  expertise  or  for 
whom  urgency  is  not  an  issue,  probably  the  most  useful.  But  suppose  that  the  recipient  had  some 
additional  resources  and  an  interest  in  pursuing  an  alternate  development  path,  possibly 
involving  triangulation  or  cross-correlation  of  data  between  platforms?  Would  it  not  be 
worthwhile  to  exchange  sonar  data  at  a  level  at  which  the  additional  processing  would  still  be 
feasible?  The  answer  would  depend  on  the  availability  of  sufficient  communication  bandwidth 
and  processing  capability  to  deal  with  the  data  and  the  value  of  the  expected  result.  In  order  to 
investigate  the  value  of  coalition  sonar  track  sharing,  it  is  useful  to  consider  the  influence  of 
shared  sonar  track  data,  and  how  the  value  of  that  shared  data  changes  with  the  level  of 
refinement  of  that  data  and  the  tools  and  training  available  to  the  sonar  operator. 

1 .3  Virtual  Battle  Experiments 

Under  its  mandate  to  investigate  common  areas  of  research  regarding  maritime  command, 
control  and  information  management,  Maritime  Systems  Group  Technical  Panel  1  of  The 
Technical  Cooperation  Panel  (TTCP  MAR  TP-1)  has  been  conducting  Virtual  Battle 
Experiments  (VBEs)  using  a  simulated  maritime  environment  to  investigate  the  benefits  of 
Network  Enabled  Capability  (NEC)  [4],  The  simulations  have  been  constructed  by  assembling 
Virtual  Maritime  Systems  Architecture1  (VMSA)  based  federates  into  a  High  Level  Architecture 
(HLA)  compliant  federation  [5].  Each  federate  provides  component  or  subsystem  capabilities  of 
varying  scope,  allowing  the  simulations  to  be  tailored  to  the  needs  of  the  experiment. 


1  VMSA  was  developed  by  the  Australian  Defence  Science  and  Technology  Organization.  The  architecture  and 
many  simulation  components,  called  federates,  were  made  available  to  DRDC  Atlantic  through  a  Memorandum  of 
Understanding  between  Canada  and  Australia,  Subsidiary  Arrangement  18. 


VMSA-based  simulations  are  time  managed,  allowing  them  to  be  well  controlled  and  repeatable. 
By  taking  advantage  of  this  feature,  the  results  of  multiple  independent  runs  of  an  experiment 
using  different  human  operators  can  be  analyzed  to  compile  statistically  relevant  answers  to  the 
experimental  objective.  The  distributed  and  synchronous  nature  of  this  architecture  is  also  well 
suited  to  the  logging  and  subsequent  analysis  of  experimental  data. 

2.  Coalition  Sonar  Track  Sharing 

The  Coalition  Sonar  Track  Sharing  (CoaSTS)  program  is  a  series  of  controlled,  repeatable, 
human-in-the-loop  experimentations  to  investigate  the  influence  of  shared  sonar  track  data  at 
various  levels  of  development.  The  program  uses  a  VMSA-based  maritime  simulation  in  which 
a  sonar  operator  is  tasked  to  use  organic,  and  in  some  cases  nonorganic,  broadband  passive  sonar 
tracks  to  monitor  local  vessel  traffic.  The  quality  of  the  local  operating  picture  is  determined  by 
the  degree  to  which  the  operator  has  successfully  identified  and  associated  the  multiple, 
temporally  distinct  sonar  track  segments  originating  from  each  target  vessel.  Multiple 
independent  sessions  of  each  configuration  are  run  using  different  operators  and  a  limited 
number  of  traffic  scenarios.  Statistically  relevant  comparisons  can  then  be  made  between  the 
results  of  various  configurations. 

The  program  uses  the  experimental  infrastructure  shown  in  Figure  2  to  address  the  following 
question: 

What  is  the  influence  of  shared  sonar  track  data,  and  how  does  the  value  of  that  shared  data 
change  with  the  level  of  refinement  of  that  data  and  the  tools  and  training  available  to  the  sonar 
operator? 

Measurements  using  the  following  6  configurations  are  sufficient  to  make  the  most  relevant 
comparisons. 

1.  Basic  organic.  The  basic  configuration  of  the  passive  sonar  system  contains  only 
ownship  track  and  chart  displays.  The  organic  passive  sonar  produces  sonar  track 
segments  that  appear  on  the  track  display.  The  operator  is  tasked  to  develop  a  local 
operating  picture  by  fusing  the  track  segments  into  longer,  master  tracks.  The  position  of 
the  operator’s  vessel,  ownship,  is  shown  on  a  chart  display.  (White  portions  of  Figure  2) 

2.  Basic  organic  and  nonorganic.  Sonar  data  from  a  similarly  equipped  allied  ship  are 
also  available  on  a  separate  display.  The  sonar  operator  can  associate  the  organic  or 
nonorganic  sonar  track  segments  into  master  tracks  but  cannot  intermix,  other  than 
cognitively,  the  organic  and  nonorganic  data.  The  positions  of  both  ships  are  shown  on 
the  chart  display.  (White  and  green  portions  of  Figure  2) 

3.  Basic  organic  and  nonorganic  with  triangulation.  The  sonar  operator  has  access  to 
both  organic  and  nonorganic  sonar  data  and  a  triangulation  tool  with  which  to  evaluate 
associations  between  the  two.  The  operator  continues  to  be  responsible  for  developing 
the  local  operating  picture  by  associating  sonar  track  segments  but  now  has  a  tool  with 


Figure  2.  The  CoaSTS  experimentation  infrastructure.  The  basic  organic  components  are  white.  The  basic 
nonorganic  components  are  green ,  the  triangulation  components  are  yellow,  the  organic  TMA 
components  are  pink,  and  the  nonorganic  TMA  components  are  blue.  Those  components  that  have  a 
highlighted  border  require  operator  interaction. 


which  to  evaluate  the  relationship  between  organic  and  nonorganic  tracks.  (White,  green 
and  yellow  portions  of  Figure  2) 

4.  Basic  organic  and  nonorganic  with  triangulation  and  organic  TMA.  The  sonar 
operator  has  a  TMA  tool  with  which  to  convert  the  bearings-only  master  tracks  shown  on 
the  ownship  track  display  into  latitude-longitude  target  vessel  position  estimates  on  the 
chart  display.  The  triangulation  tool  continues  to  be  available,  and  the  operator  continues 
to  be  responsible  for  developing  the  local  operating  picture  by  associating  sonar  track 
segments.  (White,  green,  yellow  and  pink  portions  of  Figure  2) 

5.  Basic  organic  and  nonorganic  with  triangulation  and  dual  TMA.  A  triangulation  tool 
and  TMA  tools  are  available  for  use  with  both  organic  and  nonorganic  sonar  tracks.  The 
operator  continues  to  be  responsible  for  developing  the  local  operating  picture  by 
associating  sonar  track  segments.  (All  of  Figure  2) 

6.  Basic  organic  and  nonorganic  with  dual  TMA.  TMA  tools  are  available  for  use  with 
both  organic  and  nonorganic  sonar  tracks  but  the  triangulation  tool  is  not  available.  The 


operator  continues  to  be  responsible  for  developing  the  local  operating  picture  by 
associating  sonar  track  segments.  (White,  green  and  blue  portions  of  Figure  2) 


Since  configuration  2  is  the  first  level  at  which  nonorganic  data  is  available,  it  represents  the 
lowest  level  at  which  sonar  data  might  be  shared  in  this  experimental  program.  Comparisons  of 
configurations  1  and  2  provide  a  baseline  for  the  following  experiments  by  identifying  the 
minimum  configuration  in  which  the  sharing  of  sonar  data  could  provide  some  level  of  benefit. 
The  degree  of  benefit  was  determined  by  experiment,  although  the  lack  of  tools  beyond  the 
operator’s  cognition  is  a  severe  constraint. 

The  operator  can  use  the  triangulation  tool  in  scenario  3  to  translate  pairs  of  bearings-only  tracks 
into  estimates  of  target  vessel  position  tracks  on  the  display  chart.  This  could  be  used  to  evaluate 
the  likelihood  that  a  pair  of  organic  and  nonorganic  sonar  tracks  shares  a  common  origin.  Track 
segment  association  continues  to  be  relevant  since  the  duration  of  the  shortest  bearings-only 
track  would  determine  the  duration  of  the  position  tracks.  Although  gaps  in  a  bearings-only 
track  would  produce  gaps  in  the  corresponding  position  track,  apparent  track  continuity  across 
both  gaps  would  strongly  suggest  that  the  tracks  share  a  common  origin. 

The  availability  of  organic  TMA  in  scenario  4  allows  the  operator  to  compare  position 
information  developed  from  an  organic  bearings-only  track  with  that  developed  from 
triangulation  of  that  track  with  a  nonorganic  bearings-only  track.  This  provides  a  second  venue 
for  evaluating  whether  the  pair  of  tracks  shares  a  common  origin.  Track  segment  association 
continues  to  be  relevant  since  the  duration  of  the  organic  track  has  the  potential  to  influence  the 
TMA  results.  In  scenario  5,  this  ability  to  compare  TMA  and  triangulation  solutions  is  extended 
as  nonorganic  TMA  becomes  available.  In  scenario  6,  the  triangulation  tool  is  not  available  but, 
by  making  use  of  the  dual  TMA  tools,  the  operator  can  continue  to  compare  pairs  of  organic  and 
nonorganic  bearings-only  tracks  on  the  chart  display. 

In  each  of  the  above  cases,  the  abilities  of  the  sonar  operator  are  varied  through  the  incremental 
changes  in  the  available  processing  tools.  Meaningful  comparisons  can  therefore  be  made 
between  configurations  to  determine  the  relative  influence  of  the  tools  on  the  effectiveness  of  the 
operator.  Comparisons  can  also  be  made  among  multiple  implementations  of  each  tool. 

3.  Virtual  Battle  Experiment  CA-1 

3.1  Objectives 

VBE  CA-1  was  an  implementation  of  the  first  two  CoaSTS  configurations  and  was  intended  to 
provide  a  baseline  for  future  data  sharing  experiments.  The  experiment  was  also  motivated  by  an 
interest  in  the  types  of  algorithms  used  to  associate  passive  sonar  tracks  [6,7,8],  The  objectives 
of  this  experiment  were  a  hypothesis  test  and  a  discovery  exercise: 

1 .  Sonar  track  sharing  between  coalition  partners  is  beneficial  even  when  the  data  can  only  be 
shared  as  an  independent  display. 

2.  What  is  the  rationale  used  by  a  sonar  operator  to  make  association  and  disassociation 
decisions? 
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Figure  3.  The  Horizon  chart  display  showed  the  navigable  waters  of  the  strait  and  the  locations  of  the  ownship  and 
the  allied  ship. 


The  hypothesis  test  was  to  be  addressed  by  comparing  quantitative  metrics  describing  the 
outcome  of  the  two  experimental  configurations  while  the  discovery  exercise  was  to  be 
addressed  by  cataloguing  and  summarizing  operator  responses. 

3.2  Design  and  Implementation 

In  the  baseline  scenario  the  ownship  was  a  frigate  using  a  towed  array  sonar  to  monitor  vessel 
traffic  in  a  narrow  strait.  A  similarly  equipped  allied  ship  patrolled  the  other  side  of  the  strait. 

The  sonar  operator  was  provided  with  a  chart  display,  shown  in  Figure  3,  showing  the  position  of 
the  ownship  and  the  allied  ship  in  the  strait  and  a  track  display,  shown  in  Figure  4,  showing  the 
sonar  track  segments  produced  by  the  ownship  sonar.  When  shared  data  was  available,  sonar 
track  segments  from  the  allied  ship  were  provided  on  a  separate  and  independent  display.  The 
sonar  tracks  provided  bearing  information  only  and  were  broken  up  in  time  due  to  the  effects  of 
the  local  environment  and  interference  between  the  acoustic  emissions  of  the  target  vessels.  The 
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Figure  4.  The  Horizon  time-bearing  display  showed  sonar  track  segments  produced  by  the  ownship  sonar. 


acoustic  signals  were  not  scattered,  reflected  or  refracted,  so  each  vessel  in  the  strait  produced  no 
more  than  one  track  at  any  time.  Each  sonar  provided  full  coverage  of  the  travelled  portion  of 
the  strait.  The  task  of  the  sonar  operator  was  to  develop  as  complete  an  operational  picture  as 
possible  by  associating  the  sonar  track  segments  into  longer,  fused  tracks. 

The  core  of  the  experimentation  infrastructure  was  the  VMSA  federation  running  the  simulation. 
The  federation  was  made  up  of  a  minimum  number  of  federates,  or  software  modules,  each 
representing  components  of  a  shipboard  system.  Gameboard,  an  automated  helm  federate  that 
executed  script  files  produced  by  Scenario  Generator  [9],  controlled  the  movement  of  the  vessels 
in  the  strait.  The  helm  commands  were  translated  into  position,  course  and  speed  by  the  motion 
federate  while  a  sonar  federate  of  our  own  design  observed  the  positions  and  types  of  the  vessels 
in  the  strait  and  produced  sonar  track  segments  for  display  to  the  sonar  operator.  The  sonar 
federate  also  produced  a  log  file  recording  relevant  characteristics  of  the  tracks  that  it  produced. 
Five  instances  of  Horizon,  a  track  data  hosting  and  management  system,  were  used  to  construct 
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Figure  5.  Physical  layout  and  distribution  of  the  experimentation  infrastructure.  Each  dashed  box  represents  a 
single  computer.  All  subject  displays  are  individual  instances  of  Horizon. 


two  operator  positions  as  shown  in  Figure  5,  one  with  two  displays  for  the  solo  case  and  another 
with  three  displays  for  the  coalition  case.  This  permitted  two  operators  to  interact  with  a  given 
scenario  simultaneously  and  independently.  Horizon  also  produced  its  own  text  and  binary  log 
files. 

The  ownship  heading  was  shown  in  the  Horizon  track  display  in  Figure  4  as  a  series  of  filled 
grey  circles.  Sonar  track  segments  were  shown  as  a  series  of  open  yellow  circles  terminated  by  a 
question  mark.  The  Active  Fusion  tool  in  the  lower  left  comer  could  be  used  to  associate  one  or 
more  sonar  track  segments  and/or  fused  tracks  into  a  fused  track  by  selecting  the  tracks  to  be 
associated  and  then  clicking  on  the  green  checkmark.  Tracks  that  had  been  fused  would  then  be 
removed  from  the  display  and  replaced  by  a  series  of  lightning  bolt  symbols  indicating  the  new 
fused  track.  The  Active  Fusion  tool  could  also  be  used  to  disassociate  a  fused  track  back  into  its 
component  track  segments. 

An  ideal  time  to  query  an  operator  as  to  the  rationale  for  making  an  association  or  disassociation 
was  immediately  after  the  decision  had  been  made.  This  was  done  using  a  program  called 
EnterReason  to  detect  mouse  clicks  in  a  predefined  region  of  the  screen  and  then  present  a  list  of 


Figure  6.  The  EnterReason  popup  would  appear  whenever  an  association  or  disassociation  was  made. 


possible  reasons  as  a  popup  query  on  the  screen.  The  list,  shown  in  Figure  6,  included  an  ‘other’ 
option  which  would  open  a  text  box  to  accept  a  freeform  response.  The  time  and  nature  of  the 
decision  and  operator’s  response  were  logged  to  a  file. 

A  set  of  quantitative  metrics  was  used  to  make  clear  and  unambiguous  comparisons  between  the 
coalition  case,  which  included  shared  sonar  data  from  the  allied  ship,  and  the  solo  case,  which 
did  not  include  shared  data.  The  metrics  included  Picture  Clarity,  Track  Continuity,  Association 
Continuity,  Association  Completeness,  Association  Correctness  and  Association  Delay.  In  spite 
of  multiple  prior  test  runs,  a  misunderstanding  with  respect  to  the  Horizon  text  logging  meant 
that  only  a  partial  set  of  association  information  was  available  for  analysis2.  The  full  set  of 
information  was  recorded  in  the  Horizon  binary  log  file;  this  data  will  be  analyzed  in  the  future. 
The  only  metrics  suitable  for  use  with  the  reduced  data  set  were  the  Association  Correctness,  a 
ratio  of  the  number  of  correct  associations  to  the  total  number  of  associations,  and  the 
Association  Delay,  a  measure  of  the  time  lag  between  an  association  decision  and  the  latest 
initiation  time  of  the  component  tracks. 


2  The  Horizon  text  log  was  a  periodic  system  state  log  that  recorded  the  status  of  all  active  tracks  every  15  seconds. 
In  testing,  this  was  adequate  for  the  complete  recording  of  the  operator’s  actions.  In  practice,  decisions  were  made 
more  rapidly  than  anticipated,  and  some  associations  were  made  involving  only  track  segments  that  were  no  longer 
being  actively  updated.  Event-based  logging  would  have  provided  a  better  record  in  this  case. 


3.3  Execution 


Although  much  of  the  work  of  this  experiment  centred  on  the  development  of  a  maritime 
simulation,  it  is  the  interaction  of  the  human  subject  with  the  simulation  that  is  of  primary 
interest.  The  simulation  is  a  tool,  a  stand-in  for  the  real  world.  The  ideal  candidates  for  these 
naval  sonar  experiments  are,  ideally,  experienced  naval  sonar  operators.  A  statistically  relevant 
experiment,  however,  requires  a  significant  number  of  operators  and  there  are  several  challenges 
inherent  in  their  recruitment.  In  order  to  limit  the  complexity  of  this  experiment,  it  was  decided 
to  use  technically  competent  in-house  personnel  in  this  experiment  and,  based  on  those  results, 
recruit  an  appropriately  sized  pool  of  naval  sonar  operators  for  later  experiments. 

Eight  volunteers  participated  in  the  experimentation  sessions,  each  subject  taking  the  positions  of 
solo  operator  and  coalition  operator  once  and  seeing  a  different  scenario  each  time.  A  typical 
experimentation  day  included  45  minutes  of  training  followed  by  a  break,  then  a  2  hour 
experimentation  session.  A  second  2  hour  experimentation  session  was  held  after  lunch, 
followed  by  a  break.  The  day  ended  with  a  45  minute  exit  interview,  which  provided  an 
opportunity  for  the  subjects  to  describe  their  previous  experience  with  sonar  and  provide 
feedback  on  the  experimentation  sessions.  The  scenario  used  for  training  was  never  used  in  an 
experimentation  session. 

3.4  Analysis 

Automated  analysis  software  was  developed  to  use  the  metrics  described  above  to  produce 
performance  summaries  for  each  of  the  subjects  and  the  experimental  sessions.  The  analysis  was 
done  in  several  steps,  each  of  which  produced  a  log  file  for  verification  and  potential  later 
analysis.  A  summary  of  all  of  the  sessions  was  also  produced.  The  summary  file  was  further 
analyzed  to  produce  the  overall  results. 

In  the  16  experimentation  sessions,  each  sonar  produced  an  average  of  157  track  segments  from 
the  9  observable  vessels.  The  Horizon  text  log  file  did  not  record  all  of  the  track  associations 
made  by  an  operator,  rather,  it  recorded  only  those  that  were  later  updated  with  new  track  data. 
76%  of  the  track  associations  could  be  recovered,  but  only  67%  had  sufficient  information  to  be 
analyzable. 

In  both  the  solo  and  coalition  cases,  the  mean  Association  Correctness  for  track  segments 
originating  from  an  individual  vessel  was  78%  for  solo  tracks  and  96%  for  joint  tracks.  Solo 
tracks  are  segments  that  originated  from  a  single  distinct  target  vessel,  while  joint  tracks 
represent  multiple  vessels  that  were  indistinguishably  close  in  bearing.  The  higher  score  in  the 
joint  track  case  is  not  surprising  since  each  of  the  associated  sets  needed  only  a  single  common 
target  to  be  correct  and  each  joint  track  identified  several  targets.  The  mean  Association 
Correctness  scores  in  the  coalition  case  were  4%  and  5%  better  than  in  the  solo  case,  for  the  solo 
track  and  joint  track  cases  respectively. 

The  association-weighted  mean  Association  Delay  was  804  seconds  in  the  coalition  case  and  908 
in  the  solo  case,  with  respective  standard  deviations  of  54.3  and  60.9  seconds  for  the  means  and 
963  and  1192  seconds  for  single  observations.  Although  the  mean  association  delay  was  11% 


lower  in  the  coalition  case,  the  level  of  significance  was  only  90%  [10].  If  the  full  set  of  fusions 
had  been  available,  the  level  of  significance  would  rise  to  93%,  which  is  still  not  sufficient  for  a 
conclusive  result.  Based  on  these  statistics,  6  additional  subjects  would  have  been  needed  to 
reach  at  least  a  95%  level  of  significance. 

The  popup  query  did  not  appear  to  be  a  very  effective  method  of  obtaining  the  decision  rationale 
information  that  we  were  looking  for,  possibly  because  the  operators  saw  the  query  as  a  multiple 
choice  question  in  which  at  least  one  of  the  presented  answers  is  correct,  possibly  because  the 
freeform  nature  of  the  ‘other’  choice  left  the  responses  too  open,  or  possibly  because  of  the 
limited  expertise  of  the  operators.  The  most  frequently  cited  option  was  ‘Bearing  Continuity’. 
The  most  insightful  freeform  responses  were  ‘Bearing  Rate’,  ‘Track  Continuity’  and  ‘Chart 
Position’. 

In  the  exit  interviews,  most  subjects  indicated  low  to  medium-high  satisfaction  with  their 
development  of  the  local  operating  picture  and  appeared  to  be  comfortable  with,  but  not 
confident  in,  the  results.  The  challenge  of  having  to  think  in  bearing-space  instead  of  the  more 
familiar  2-dimensional  distance  space  was  cited  as  a  significant  difficulty.  The  most  frequent 
request  was  for  a  tool  with  which  to  triangulate  pairs  of  organic  and  nonorganic  bearings-only 
sonar  tracks  into  positional  tracks. 

4.  Concluding  Remarks 

Based  on  the  particular  scenarios  considered  in  this  experiment,  the  results  of  the  quantitative 
analysis  provided  slight  but  not  conclusive  evidence  of  an  improvement  in  operator  performance 
due  to  the  presence  of  shared  data.  The  association  decision  rationales  indicated  by  the  operators 
were  interesting,  but  not  as  insightful  as  had  been  hoped. 

In  the  follow-on  experiment  it  would  be  useful  to  provide  the  same  level  of  passive  sonar  data  in 
conjunction  with  a  triangulation  tool,  so  that  the  recipient  could  triangulate  passive  sonar  tracks 
from  both  vessels  to  estimate  the  positional  tracks  of  the  target  vessels.  In  this  experiment  each 
coalition  vessel  had  full  coverage  of  the  travelled  portion  of  the  strait  but  from  differing 
perspectives.  In  future  experiments  this  might  be  reduced  to  variable  degrees  of  partial  coverage. 
The  results  of  this  experiment  provide  a  baseline  for  these  and  other  follow-on  experiments. 
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Coalition  Sonar  Track  Sharing  (CoaSTS) 


•  CoaSTS  is  a  series  of  controlled,  repeatable, 
human-in-the-loop  experiment  configurations  to 
investigate  the  influence  of  shared  sonar  track  data 
at  various  levels  of  development. 

•  It  is  a  VMSA/HLA-based  simulation  that  requires  a 
human  operator  to  use  passive  sonar  tracks  to 
monitor  local  vessel  traffic. 


Compares  the  results  of  multiple  independent 
operators  running  multiple  independent  sessions  of 
a  limited  number  of  scenarios  to  produce 
statistically  relevant  results. 
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Coalition  Sonar  Track  Sharing 
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Coalition  Sonar  Track  Sharing 
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VBE  CA-1  Objectives 


•  VBE  CA-1  uses  the  first  two  experimental 
configurations,  with  neither  triangulation  nor  TMA, 
to  address  the  following  objectives: 

•  Hypothesis  Testing:  Sonar  track  sharing  among 
coalition  partners  is  beneficial  even  when  the  data 
can  only  be  shared  as  an  independent  display. 


Discovery:  What  is  the  rationale  used  by  a  sonar 
operator  to  make  association  and  disassociation 
decisions? 
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VBE  CA-1  Scenario 


VBE  CA-1  Track  Display 
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Recording  the  Decision  Rationale 


•  This  query  appears 
whenever  the 
operator  makes  a 
(dis)association  or 
decision. 


The  ‘other’  choice 
opens  a  text  box  to 
record  a  free-form 
response  of  the 
operator’s  choice. 
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VBE  CA-1  Configuration 
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Findings 


n$J 

Findings 

•  The  presence  of  low-level  non-organic  data 
correlated  with  a 

slight  increase  in  Association  Correctness  and  a 
small  reduction  in  the  Association  Delay. 

ft 

-  Correctness  =  number  of  correct  associations 

total  number  of  associations  made 

-  Delay  =  track  fusion  time  -  latest  track  initiation  time 

ft 

•  The  Decision  Rationale  test  was  inconclusive. 
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Conclusions 

•  A  baseline  has  been  established  for  the  follow-on 
experiments. 


-  More  operators  will  be  required  for  statistical 
significance. 

-  An  experimentation  infrastructure  and 
procedures  have  been  established. 


-  A  more  accurate  sonar  simulation  is  required 


Future  Work 


The  next  experiment  will  include  triangulation. 
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